{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Clone Portal users, groups and content\n",
    "\n",
    "This sample notebook can be used for cloning a portal, from say, a staging to a production environment. It clones the users, groups and the content. It does not copy over services though, and works at the tier of portal items.\n",
    "\n",
    "**Note**: To use this notebook as a Python script, check out the accompanying [SDK GitHub](https://github.com/Esri/arcgis-python-api) repository. Running this as a script from a Python IDE allows you to set breakpoints, debug, and inspect the script when an exception is raised."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "toc": true
   },
   "source": [
    "<h1>Table of Contents<span class=\"tocSkip\"></span></h1>\n",
    "<div class=\"toc\"><ul class=\"toc-item\"><li><span><a href=\"#Clone-Portal-users,-groups-and-content\" data-toc-modified-id=\"Clone-Portal-users,-groups-and-content-1\"><span class=\"toc-item-num\">1&nbsp;&nbsp;</span>Clone Portal users, groups and content</a></span><ul class=\"toc-item\"><li><span><a href=\"#Define-the-source-and-target-portals\" data-toc-modified-id=\"Define-the-source-and-target-portals-1.1\"><span class=\"toc-item-num\">1.1&nbsp;&nbsp;</span>Define the source and target portals</a></span></li></ul></li><li><span><a href=\"#Users\" data-toc-modified-id=\"Users-2\"><span class=\"toc-item-num\">2&nbsp;&nbsp;</span>Users</a></span><ul class=\"toc-item\"><li><ul class=\"toc-item\"><li><span><a href=\"#Remove-existing-users-from-target-portal\" data-toc-modified-id=\"Remove-existing-users-from-target-portal-2.0.1\"><span class=\"toc-item-num\">2.0.1&nbsp;&nbsp;</span>Remove existing users from target portal</a></span></li><li><span><a href=\"#Copy-Users\" data-toc-modified-id=\"Copy-Users-2.0.2\"><span class=\"toc-item-num\">2.0.2&nbsp;&nbsp;</span>Copy Users</a></span></li></ul></li></ul></li><li><span><a href=\"#Groups\" data-toc-modified-id=\"Groups-3\"><span class=\"toc-item-num\">3&nbsp;&nbsp;</span>Groups</a></span><ul class=\"toc-item\"><li><span><a href=\"#Copy-Groups\" data-toc-modified-id=\"Copy-Groups-3.1\"><span class=\"toc-item-num\">3.1&nbsp;&nbsp;</span>Copy Groups</a></span></li></ul></li><li><span><a href=\"#Items\" data-toc-modified-id=\"Items-4\"><span class=\"toc-item-num\">4&nbsp;&nbsp;</span>Items</a></span><ul class=\"toc-item\"><li><span><a href=\"#For-each-user-create-a-mapping-of-itemId-to-the-Item\" data-toc-modified-id=\"For-each-user-create-a-mapping-of-itemId-to-the-Item-4.1\"><span class=\"toc-item-num\">4.1&nbsp;&nbsp;</span>For each user create a mapping of itemId to the <code>Item</code></a></span></li><li><span><a href=\"#Prepare-sharing-information-for-each-item\" data-toc-modified-id=\"Prepare-sharing-information-for-each-item-4.2\"><span class=\"toc-item-num\">4.2&nbsp;&nbsp;</span>Prepare sharing information for each item</a></span><ul class=\"toc-item\"><li><span><a href=\"#Print-a-mapping-of-item-and-its-group-membership\" data-toc-modified-id=\"Print-a-mapping-of-item-and-its-group-membership-4.2.1\"><span class=\"toc-item-num\">4.2.1&nbsp;&nbsp;</span>Print a mapping of item and its group membership</a></span></li></ul></li><li><span><a href=\"#Copy-Items\" data-toc-modified-id=\"Copy-Items-4.3\"><span class=\"toc-item-num\">4.3&nbsp;&nbsp;</span>Copy Items</a></span></li><li><span><a href=\"#Establish-relationship-between-items\" data-toc-modified-id=\"Establish-relationship-between-items-4.4\"><span class=\"toc-item-num\">4.4&nbsp;&nbsp;</span>Establish relationship between items</a></span></li><li><span><a href=\"#Conclusion\" data-toc-modified-id=\"Conclusion-4.5\"><span class=\"toc-item-num\">4.5&nbsp;&nbsp;</span>Conclusion</a></span></li></ul></li></ul></div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import libraries\n",
    "from arcgis.gis import GIS\n",
    "from IPython.display import display\n",
    "from getpass import getpass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define the source and target portals\n",
    "To start with, define the source and target portals. Connect to them using accounts with administrative privileges:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "········\n",
      "········\n"
     ]
    }
   ],
   "source": [
    "source_password = getpass()\n",
    "target_password = getpass()\n",
    "source = GIS(\"source portal url\", username, source_password)\n",
    "target = GIS(\"target portal url\", username, target_password)\n",
    "target_admin_username = 'admin'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Users\n",
    "List the users in the source and target portals. We do not want to copy over system accounts since those would be available in the target portal as well. Hence, filter the search by negating any account that starts with 'esri_'. We also do not want to copy over the [initial administrator account](http://server.arcgis.com/en/portal/latest/administer/linux/about-the-initial-administrator-account.htm) as one would be present in the target as well. Hence, negate the account that starts with `admin` which happens to be the administrator account on source portal."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "brown.rogers\t:\torg_user\n",
      "davis.reed\t:\torg_admin\n",
      "johnson.stewart\t:\torg_user\n",
      "jones.morris\t:\torg_user\n",
      "miller.cook\t:\torg_publisher\n",
      "moore.bell\t:\torg_publisher\n",
      "project_archiver\t:\torg_user\n",
      "smith.collins\t:\torg_admin\n",
      "taylor.murphy\t:\torg_publisher\n",
      "williams.sanchez\t:\torg_user\n",
      "wilson.morgan\t:\torg_publisher\n"
     ]
    }
   ],
   "source": [
    "#!esri_ & !admin\n",
    "source_users = source.users.search('!esri_ & !admin')\n",
    "for user in source_users:\n",
    "    print(user.username + \"\\t:\\t\" + str(user.role))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the number of users to migrate:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "11"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(source_users)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the list of users already present in the target portal. Similar to earlier, filter out system and initial administrator accounts. The name of the admin account on target portal is `admin` as well in this example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<User username:arcgis_python_api>, <User username:publisher1>]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# filter out system and initial administrator accounts\n",
    "target_users = target.users.search('!esri_ & !admin & !system_publisher')\n",
    "target_users"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If users found on source portal were already in the target portal, run the following code to delete them. You can choose to not delete them as well.\n",
    "\n",
    "### Remove existing users from target portal\n",
    "If you want to clean up the target portal except for the initial administrator account, run the cell below. As you delete, you may opt to assign their content to the initial administrator account."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "for source_user in source_users:\n",
    "    try:\n",
    "        target_user = target.users.get(source_user.username)\n",
    "        if target_user is not None:\n",
    "            print('Deleting user: ' + target_user.fullName)\n",
    "            target_user.reassign_to(target_admin_username)\n",
    "            target_user.delete()\n",
    "    except:\n",
    "        print('User {} does not exist in Target Portal'.format(source_user.username))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Copy Users\n",
    "Create a function that will accept connection to the target portal, `User` objects from source portal and password to create users with. In addition to creating the users, this function will set their access, description, tags and other similar properties from source. If a user by the same name already exists in the target portal (possible if you opted not to clean out the target portal) then this function prints out an error message."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "def copy_user(target_portal, source_user, password):\n",
    "    # See if the user has firstName and lastName properties\n",
    "    try:\n",
    "        first_name = source_user.firstName\n",
    "        last_name = source_user.lastName\n",
    "    except:\n",
    "        # if not, split the fullName\n",
    "        full_name = source_user.fullName\n",
    "        first_name = full_name.split()[0]\n",
    "        try:\n",
    "            last_name = full_name.split()[1]\n",
    "        except:\n",
    "            last_name = 'NoLastName'\n",
    "\n",
    "    try:\n",
    "        # create user\n",
    "        target_user = target_portal.users.create(source_user.username, password, first_name, \n",
    "                                                 last_name, source_user.email, \n",
    "                                                 source_user.description, source_user.role)\n",
    "\n",
    "        # update user properties\n",
    "        target_user.update(source_user.access, source_user.preferredView,\n",
    "                           source_user.description, source_user.tags, \n",
    "                           source_user.get_thumbnail_link(),\n",
    "                           culture=source_user.culture, region=source_user.region)\n",
    "        return target_user\n",
    "    \n",
    "    except Exception as Ex:\n",
    "        print(str(Ex))\n",
    "        print(\"Unable to create user \"+ source_user.username)\n",
    "        return None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For each user in source portal, make a corresponding user in target portal. In this sample, we provide a common password to all users `TestPassword@123` as we are creating users off the built-in identity store. If you are creating users off your enterprise identity store, you can ignore the `password` parameter and use the `provider` and `idp_username` parameters as explained in the [API reference doc](https://developers.arcgis.com/python/api-reference/arcgis.gis.toc.html#arcgis.gis.UserManager.create)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating user: brown.rogers\n",
      "Creating user: davis.reed\n",
      "Creating user: johnson.stewart\n",
      "Creating user: jones.morris\n",
      "Creating user: miller.cook\n",
      "Creating user: moore.bell\n",
      "Creating user: project_archiver\n",
      "Creating user: smith.collins\n",
      "Creating user: taylor.murphy\n",
      "Creating user: williams.sanchez\n",
      "Creating user: wilson.morgan\n"
     ]
    }
   ],
   "source": [
    "for user in source_users:\n",
    "    print(\"Creating user: \" + user.username)\n",
    "    copy_user(target, user, 'TestPassword@123')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Verify that users have been added to target portal:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<User username:admin>,\n",
       " <User username:arcgis_python_api>,\n",
       " <User username:brown.rogers>,\n",
       " <User username:davis.reed>,\n",
       " <User username:esri_boundaries>,\n",
       " <User username:esri_demographics>,\n",
       " <User username:esri_livingatlas>,\n",
       " <User username:esri_nav>,\n",
       " <User username:johnson.stewart>,\n",
       " <User username:jones.morris>,\n",
       " <User username:miller.cook>,\n",
       " <User username:moore.bell>,\n",
       " <User username:project_archiver>,\n",
       " <User username:publisher1>,\n",
       " <User username:smith.collins>,\n",
       " <User username:system_publisher>,\n",
       " <User username:taylor.murphy>,\n",
       " <User username:williams.sanchez>,\n",
       " <User username:wilson.morgan>]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "target_users = target.users.search()\n",
    "target_users"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Thus, users have been successfully added to the target portal"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Groups"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "List the groups in the source and target portals. Similar to how we searched for users, we will ignore the system created and default groups as they would be available on the target portal as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<Group title:\"Central Services\" owner:admin>,\n",
       " <Group title:\"Compliance\" owner:admin>,\n",
       " <Group title:\"Customer Service, Finance, Billing and Accounting\" owner:admin>,\n",
       " <Group title:\"Demographic Content\" owner:admin>]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# filter out system created groups\n",
    "source_groups = source.groups.search(\"!owner:esri_* & !Basemaps\")\n",
    "source_groups"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<Group title:\"Featured Maps and Apps\" owner:admin>]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "target_groups = target.groups.search(\"!owner:esri_* & !Basemaps\")\n",
    "target_groups"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If any of the groups from source are already in the target, run the following code to delete them. If the group belongs to any of default user accounts, don't delete it. This step is optional, you may choose to not delete those groups if you prefer to retain them as is."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "for tg in target_groups:\n",
    "    for sg in source_groups:\n",
    "        if sg.title == tg.title and (not tg.owner.startswith('esri_')):\n",
    "            print(\"Cleaning up group {} in target Portal...\".format(tg.title))\n",
    "            tg.delete()\n",
    "            break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Copy Groups\n",
    "\n",
    "Let us create a function that will clone the groups one at a time. As you call this function in a loop for each group, it reads the source group's properties, downloads thumbnail into a temporary file then creates a similar named group on target and applies those properties and thumbnail. If one of your portals is an organization on ArcGIS Online and other is an ArcGIS Enterprise, certain privacy properties need to be adapted. This function takes care of that. After creating the group, it finds which users were members of it and adds them appropriately."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "import tempfile\n",
    "\n",
    "GROUP_COPY_PROPERTIES = ['title', 'description', 'tags', 'snippet', 'phone',\n",
    "                         'access', 'isInvitationOnly']\n",
    "\n",
    "def copy_group(target, source, source_group):\n",
    "    \n",
    "    with tempfile.TemporaryDirectory() as temp_dir:\n",
    "        try:\n",
    "            target_group = {}\n",
    "\n",
    "            for property_name in GROUP_COPY_PROPERTIES:\n",
    "                target_group[property_name] = source_group[property_name]\n",
    "\n",
    "            if source_group['access'] == 'org' and target.properties['portalMode'] == 'singletenant':\n",
    "                #cloning from ArcGIS Online to ArcGIS Enterprise\n",
    "                target_group['access'] = 'public'\n",
    "\n",
    "            elif source_group['access'] == 'public'\\\n",
    "                 and source.properties['portalMode'] == 'singletenant'\\\n",
    "                 and target.properties['portalMode'] == 'multitenant'\\\n",
    "                 and 'id' in target.properties:\n",
    "                    #cloning from ArcGIS Enterprise to ArcGIS Online org\n",
    "                    target_group['access'] = 'org'\n",
    "\n",
    "            # Download the thumbnail (if one exists)\n",
    "            thumbnail_file = None\n",
    "            if 'thumbnail' in group:\n",
    "                target_group['thumbnail'] = group.download_thumbnail(temp_dir)\n",
    "\n",
    "            # Create the group in the target portal\n",
    "            copied_group = target.groups.create_from_dict(target_group)\n",
    "\n",
    "            # Reassign all groups to correct owners, add users, and find shared items\n",
    "            members = group.get_members()\n",
    "            if not members['owner'] == target_admin_username:\n",
    "                copied_group.reassign_to(members['owner'])\n",
    "            if members['users']:\n",
    "                copied_group.add_users(members['users'])\n",
    "            return copied_group\n",
    "        except:\n",
    "            print(\"Error creating \" + source_group['title'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For each group in source portal, make a corresponding group in target portal."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div class=\"9item_container\" style=\"height: auto; overflow: hidden; border: 1px solid #cfcfcf; border-radius: 2px; background: #f6fafa; line-height: 1.21429em; padding: 10px;\">\n",
       "                    <div class=\"item_left\" style=\"width: 210px; float: left;\">\n",
       "                       <a href='https://dev003735.esri.com/portal/home/group.html?id=8d318dc339bd43cc8b335614286630b6' target='_blank'>\n",
       "                        <img src='' ' class=\"itemThumbnail\">\n",
       "                       </a>\n",
       "                    </div>\n",
       "\n",
       "                    <div class=\"item_right\" style=\"float: none; width: auto; overflow: hidden;\">\n",
       "                        <a href='https://dev003735.esri.com/portal/home/group.html?id=8d318dc339bd43cc8b335614286630b6' target='_blank'><b>Central Services</b>\n",
       "                        </a>\n",
       "                        <br/>\n",
       "                        <br/><b>Summary</b>: The authoritative service catalog.   \n",
       "                        <br/><b>Description</b>: This Group contains an inventory of map services for our organization. These map services serve as building blocks for all maps and apps throughout the organization. \n",
       "                        <br/><b>Owner</b>: admin\n",
       "                        <br/><b>Created</b>: April 04, 2017\n",
       "\n",
       "                    </div>\n",
       "                </div>\n",
       "                "
      ],
      "text/plain": [
       "<Group title:\"Central Services\" owner:admin>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div class=\"9item_container\" style=\"height: auto; overflow: hidden; border: 1px solid #cfcfcf; border-radius: 2px; background: #f6fafa; line-height: 1.21429em; padding: 10px;\">\n",
       "                    <div class=\"item_left\" style=\"width: 210px; float: left;\">\n",
       "                       <a href='https://dev003735.esri.com/portal/home/group.html?id=7a54d80b397747cabea101b5477889cc' target='_blank'>\n",
       "                        <img src='' ' class=\"itemThumbnail\">\n",
       "                       </a>\n",
       "                    </div>\n",
       "\n",
       "                    <div class=\"item_right\" style=\"float: none; width: auto; overflow: hidden;\">\n",
       "                        <a href='https://dev003735.esri.com/portal/home/group.html?id=7a54d80b397747cabea101b5477889cc' target='_blank'><b>Compliance</b>\n",
       "                        </a>\n",
       "                        <br/>\n",
       "                        <br/><b>Summary</b>: Regulatory compliance tracking & reporting.\n",
       "                        <br/><b>Description</b>: A group dealing with government and industry association regulatory compliance and reporting.\n",
       "                        <br/><b>Owner</b>: admin\n",
       "                        <br/><b>Created</b>: April 04, 2017\n",
       "\n",
       "                    </div>\n",
       "                </div>\n",
       "                "
      ],
      "text/plain": [
       "<Group title:\"Compliance\" owner:admin>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div class=\"9item_container\" style=\"height: auto; overflow: hidden; border: 1px solid #cfcfcf; border-radius: 2px; background: #f6fafa; line-height: 1.21429em; padding: 10px;\">\n",
       "                    <div class=\"item_left\" style=\"width: 210px; float: left;\">\n",
       "                       <a href='https://dev003735.esri.com/portal/home/group.html?id=4b362ad621494a339da8fdfd98886e5a' target='_blank'>\n",
       "                        <img src='' ' class=\"itemThumbnail\">\n",
       "                       </a>\n",
       "                    </div>\n",
       "\n",
       "                    <div class=\"item_right\" style=\"float: none; width: auto; overflow: hidden;\">\n",
       "                        <a href='https://dev003735.esri.com/portal/home/group.html?id=4b362ad621494a339da8fdfd98886e5a' target='_blank'><b>Customer Service, Finance, Billing and Accounting</b>\n",
       "                        </a>\n",
       "                        <br/>\n",
       "                        <br/><b>Summary</b>: The Water & Sewer Billing and Collection Division manage the water and sewer accounts for residents.\n",
       "                        <br/><b>Description</b>: Typical types of users and roles that part of this group:Human Resources/PayrollAdministrative Services Officer I, IIAdministrative AideAccountantAccount ClerkPayroll ClerkDepartment AnalystRisk Management Analyst I,II,IIIAccounting TechnicianEngineering TechnicianCommon Task or Responsibilities of the users in this group:Customer ServiceReal Estate ServicesBudgets/Accounting/Long Range Financial PlanningSafety/Risk ManagementFinancingRecords Management/Clerical ServicesAuditingBill Collection and paymentConnections/DisconnectsReduced Rates / Leak AdjustmentsPool AdjustmentIrrigation and Hydrant MetersEmployee Development\n",
       "                        <br/><b>Owner</b>: admin\n",
       "                        <br/><b>Created</b>: April 04, 2017\n",
       "\n",
       "                    </div>\n",
       "                </div>\n",
       "                "
      ],
      "text/plain": [
       "<Group title:\"Customer Service, Finance, Billing and Accounting\" owner:admin>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div class=\"9item_container\" style=\"height: auto; overflow: hidden; border: 1px solid #cfcfcf; border-radius: 2px; background: #f6fafa; line-height: 1.21429em; padding: 10px;\">\n",
       "                    <div class=\"item_left\" style=\"width: 210px; float: left;\">\n",
       "                       <a href='https://dev003735.esri.com/portal/home/group.html?id=c653d6aff80e45f6a326324efdebc683' target='_blank'>\n",
       "                        <img src='' ' class=\"itemThumbnail\">\n",
       "                       </a>\n",
       "                    </div>\n",
       "\n",
       "                    <div class=\"item_right\" style=\"float: none; width: auto; overflow: hidden;\">\n",
       "                        <a href='https://dev003735.esri.com/portal/home/group.html?id=c653d6aff80e45f6a326324efdebc683' target='_blank'><b>Demographic Content</b>\n",
       "                        </a>\n",
       "                        <br/>\n",
       "                        <br/><b>Summary</b>: Esri demographic data with national coverage\n",
       "                        <br/><b>Description</b>: A catalog of Esri provided demographic content for use at the utility.\n",
       "                        <br/><b>Owner</b>: admin\n",
       "                        <br/><b>Created</b>: April 04, 2017\n",
       "\n",
       "                    </div>\n",
       "                </div>\n",
       "                "
      ],
      "text/plain": [
       "<Group title:\"Demographic Content\" owner:admin>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.display import display\n",
    "for group in source_groups:\n",
    "    target_group = copy_group(target, source, group)\n",
    "    if target_group:\n",
    "        display(target_group)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, we were able to add the groups with their thumbnails. Now let us verify that groups can be listed on the target portal:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<Group title:\"Central Services\" owner:admin>,\n",
       " <Group title:\"Compliance\" owner:admin>,\n",
       " <Group title:\"Customer Service, Finance, Billing and Accounting\" owner:admin>,\n",
       " <Group title:\"Demographic Content\" owner:admin>,\n",
       " <Group title:\"Esri Boundary Layers\" owner:esri_boundaries>,\n",
       " <Group title:\"Esri Demographic Layers\" owner:esri_demographics>,\n",
       " <Group title:\"Featured Maps and Apps\" owner:admin>,\n",
       " <Group title:\"Living Atlas\" owner:esri_livingatlas>,\n",
       " <Group title:\"Living Atlas Analysis Layers\" owner:esri_livingatlas>,\n",
       " <Group title:\"Navigator Maps\" owner:esri_nav>]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "target_groups = target.groups.search()\n",
    "target_groups"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With this part of the sample, we have successfully created users, groups and added the appropriate users to these groups. Thus, you can call the `get_members()` method one of the groups to view its members:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'admins': ['admin'],\n",
       " 'owner': 'admin',\n",
       " 'users': ['brown.rogers',\n",
       "  'johnson.stewart',\n",
       "  'taylor.murphy',\n",
       "  'smith.collins']}"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "group1 = target_groups[0]\n",
    "group1.get_members()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Items"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copying items consists of multiple steps as explained in the following section of the sample:\n",
    "\n",
    " 1. [For each user create a mapping of itemId to the `Item`](#for-each-user-create-a-mapping-of-itemid-to-the-item)\n",
    " 2. [Prepare sharing information for each item](#prepare-sharing-information-for-each-item)\n",
    "   1. [Print a mapping of item and its group membership](#print-a-mapping-of-item-and-its-group-membership)\n",
    " 3. [Copy items one by one](#copy-Items)\n",
    " 4. [Establish relationship between items](#establish-relationship-between-items)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## For each user create a mapping of itemId to the `Item`\n",
    "Do this for every folder in the user's account on the source portal"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting item ids for brown.rogers\t\tNumber of folders 1 # Number of items 3\n",
      "Collecting item ids for davis.reed\t\tNumber of folders 1 # Number of items 3\n",
      "Collecting item ids for johnson.stewart\t\tNumber of folders 1 # Number of items 3\n",
      "Collecting item ids for jones.morris\t\tNumber of folders 1 # Number of items 3\n",
      "Collecting item ids for miller.cook\t\tNumber of folders 1 # Number of items 3\n",
      "Collecting item ids for moore.bell\t\tNumber of folders 1 # Number of items 3\n",
      "Collecting item ids for project_archiver\t\tNumber of folders 7 # Number of items 18\n",
      "Collecting item ids for smith.collins\t\tNumber of folders 1 # Number of items 4\n",
      "Collecting item ids for taylor.murphy\t\tNumber of folders 1 # Number of items 3\n",
      "Collecting item ids for williams.sanchez\t\tNumber of folders 1 # Number of items 3\n",
      "Collecting item ids for wilson.morgan\t\tNumber of folders 1 # Number of items 3\n"
     ]
    }
   ],
   "source": [
    "source_items_by_id = {}\n",
    "for user in source_users:\n",
    "    num_items = 0\n",
    "    num_folders = 0\n",
    "    print(\"Collecting item ids for {}\".format(user.username), end=\"\\t\\t\")\n",
    "    user_content = user.items()\n",
    "    \n",
    "    # Get item ids from root folder first\n",
    "    for item in user_content:\n",
    "        num_items += 1\n",
    "        source_items_by_id[item.itemid] = item \n",
    "    \n",
    "    # Get item ids from each of the folders next\n",
    "    folders = user.folders\n",
    "    for folder in folders:\n",
    "        num_folders += 1\n",
    "        folder_items = user.items(folder=folder['title'])\n",
    "        for item in folder_items:\n",
    "            num_items += 1\n",
    "            source_items_by_id[item.itemid] = item\n",
    "    \n",
    "    print(\"Number of folders {} # Number of items {}\".format(str(num_folders), str(num_items)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us print the dictionary of `{item_id : Item object}`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'0c8ffe4ab7754eedbd2c514032f7e913': <Item title:\"IN\" type:Feature Service owner:williams.sanchez>,\n",
       " '0fdcdd6eed9e4e6aa83575da5c4d8ff0': <Item title:\"IN\" type:CSV owner:williams.sanchez>,\n",
       " '10eff50fda644f2fa9261ee925a24495': <Item title:\"set2_empty\" type:Map Document owner:project_archiver>,\n",
       " '14935a49a8544d8eb0eeadaa6216bbba': <Item title:\"set2_USAcities\" type:File Geodatabase owner:project_archiver>,\n",
       " '1506548f309140d89a19886913d8395a': <Item title:\"Miller Cook response locations\" type:Web Map owner:miller.cook>,\n",
       " '1cb04f79f8a74826bab704efe5206603': <Item title:\"set1_gov_sites_registration\" type:Microsoft Excel owner:project_archiver>,\n",
       " '1d8820ef73064498983af1af326ddb9d': <Item title:\"Johnson Stewart response locations\" type:Web Map owner:johnson.stewart>,\n",
       " '28984d7a69f34b51b07a087800931d6f': <Item title:\"Jones Morris response locations\" type:Web Map owner:jones.morris>,\n",
       " '297a1819ba9b43ffb8adcab6a7eebd53': <Item title:\"LA\" type:CSV owner:taylor.murphy>,\n",
       " '464fdece247e46b1ba861a7ac92701a4': <Item title:\"FL\" type:CSV owner:davis.reed>,\n",
       " '48f17e7559ca4d65b5eab5b78cabf085': <Item title:\"set1_fortune500\" type:File Geodatabase owner:project_archiver>,\n",
       " '4a6c6b94d1d840f3a19d7da922be4c82': <Item title:\"KS\" type:CSV owner:smith.collins>,\n",
       " '54cf94db12e64092bef153b9f2c07176': <Item title:\"set2_australia\" type:GeoJson owner:project_archiver>,\n",
       " '56d1921b6cfb46e89bd697668cceb708': <Item title:\"set1_mapping_tech\" type:Microsoft Powerpoint owner:project_archiver>,\n",
       " '59132371ac22462a899bcaf7b561f1cd': <Item title:\"set2_Voronoi-diagram\" type:Microsoft Word owner:project_archiver>,\n",
       " '67503e5077f240738e091e4c1da13745': <Item title:\"set3_Streets\" type:Map Document owner:project_archiver>,\n",
       " '715f8038c23b4b53a1280ae1b7b4a9bc': <Item title:\"Williams Sanchez response locations\" type:Web Map owner:williams.sanchez>,\n",
       " '71b66b8e628b46c9850cefd91f67f604': <Item title:\"NV\" type:Feature Service owner:johnson.stewart>,\n",
       " '73118cf2a8334a9884f269f81ab25c45': <Item title:\"Taylor Murphy response locations\" type:Web Map owner:taylor.murphy>,\n",
       " '734529a8daad4317bc6daedfe01fdc9f': <Item title:\"set2_catalina-points\" type:KML owner:project_archiver>,\n",
       " '75c63c8de25943f9b5ed9c477e556034': <Item title:\"set1_Chicago\" type:CSV owner:project_archiver>,\n",
       " '7be7710b5e4b44cf836139a115825b2e': <Item title:\"ID\" type:CSV owner:wilson.morgan>,\n",
       " '7c4fdc5508004253960993bbab02bf61': <Item title:\"NC\" type:CSV owner:jones.morris>,\n",
       " '7e854deebcd1442eb850a3ea84732679': <Item title:\"set1_major_cities\" type:Locator Package owner:project_archiver>,\n",
       " '819d0cbed11f4f81a98e7f388fb8eeab': <Item title:\"Smith Collins response locations\" type:Web Map owner:smith.collins>,\n",
       " '882009025e11471caad3f48389fde221': <Item title:\"FL\" type:Feature Service owner:davis.reed>,\n",
       " '8a6b1ed1b7be4cdeb298020cf5108c70': <Item title:\"set2_Chicago\" type:CSV owner:project_archiver>,\n",
       " '8d025e7368974649a3eb0691656dfa57': <Item title:\"set2_SD_crime\" type:Map Document owner:project_archiver>,\n",
       " '97d15ea9057549cdbe67344fe129c758': <Item title:\"ID\" type:Feature Service owner:wilson.morgan>,\n",
       " '9afed1a827914a53af34af490f64ecdf': <Item title:\"Moore Bell response locations\" type:Web Map owner:moore.bell>,\n",
       " 'a0929a38db3240cfaffbc254d00e1827': <Item title:\"AZ\" type:Feature Service owner:moore.bell>,\n",
       " 'a89398de64444d58a9ea6a0cc407c6b7': <Item title:\"LA\" type:Feature Service owner:taylor.murphy>,\n",
       " 'aaa6ad38f9c24b4294a4c6d0994fc7c7': <Item title:\"AR\" type:Feature Service owner:brown.rogers>,\n",
       " 'b18c614bc00d40d0a216754d1aa8b150': <Item title:\"Brown Rogers response locations\" type:Web Map owner:brown.rogers>,\n",
       " 'b8c648204c804c289930b9e121dd8fdb': <Item title:\"AZ\" type:CSV owner:moore.bell>,\n",
       " 'ba3d9b3e6793406696497be34607de44': <Item title:\"set1_GeoJson\" type:PDF owner:project_archiver>,\n",
       " 'c56e60ab29f1400fbc46cc2732724ef8': <Item title:\"Wilson Morgan response locations\" type:Web Map owner:wilson.morgan>,\n",
       " 'd65f34a670654818a010771d9a896760': <Item title:\"Davis Reed response locations\" type:Web Map owner:davis.reed>,\n",
       " 'd6818b3d16c847b5ab9986d708b992a2': <Item title:\"set2_counties\" type:Locator Package owner:project_archiver>,\n",
       " 'd8fdaa873f724c7abdca62a42de0c97a': <Item title:\"USA_cities_Fortune_500\" type:Map Document owner:project_archiver>,\n",
       " 'dee8ae7fede64f8e94c39d4f6aff8d6f': <Item title:\"KS\" type:Feature Service owner:smith.collins>,\n",
       " 'e0e716ac1d774cb190765dd7a2a7421f': <Item title:\"set1_india\" type:GeoJson owner:project_archiver>,\n",
       " 'e76f3fab66534499bdc394558a645357': <Item title:\"NV\" type:CSV owner:johnson.stewart>,\n",
       " 'ea36c5dbd4e142079c43b17cb49b4258': <Item title:\"set1_GeoJson\" type:Microsoft Word owner:project_archiver>,\n",
       " 'ebd57622fb984f1c96122c4b442ea6d7': <Item title:\"AR\" type:CSV owner:brown.rogers>,\n",
       " 'edba588da56b4a3ab2f2df576f9c6b99': <Item title:\"NH\" type:CSV owner:miller.cook>,\n",
       " 'eed82e9d95dc4ad6a0f3d21e530e0a8f': <Item title:\"Smith Collins response locations\" type:Web Map owner:smith.collins>,\n",
       " 'f3f87cb385ea4ae291fd6557338e8e0b': <Item title:\"NH\" type:Feature Service owner:miller.cook>,\n",
       " 'f9866c5a2be0428abe1c7097993d9b36': <Item title:\"NC\" type:Feature Service owner:jones.morris>}"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "source_items_by_id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prepare sharing information for each item\n",
    "Using the dictionary we created above, find to which groups are each of the items shared to."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "for group in source_groups:\n",
    "    #iterate through each item shared to the source group\n",
    "    for group_item in group.content():\n",
    "        try:\n",
    "            #get the item\n",
    "            item = source_items_by_id[group_item.itemid]\n",
    "            if item is not None:\n",
    "                if not 'groups'in item:\n",
    "                    item['groups'] = []\n",
    "                \n",
    "                #assign the target portal's corresponding group's name\n",
    "                item['groups'].append(group['title'])\n",
    "        except:\n",
    "            print(\"Cannot find item : \" + group_item.itemid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Print a mapping of item and its group membership"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "KS                                       # \n",
      "NC                                       # \n",
      "AR                                       # \n",
      "set2_catalina-points                     # \n",
      "FL                                       # \n",
      "KS                                       # \n",
      "set1_GeoJson                             # \n",
      "set2_australia                           # \n",
      "NV                                       # \n",
      "AZ                                       # \n",
      "NV                                       # \n",
      "FL                                       # \n",
      "set3_Streets                             # \n",
      "set2_counties                            # \n",
      "ID                                       # \n",
      "Brown Rogers response locations          # shared # ['Central Services']\n",
      "set1_Chicago                             # \n",
      "set2_USAcities                           # \n",
      "Jones Morris response locations          # shared # ['Customer Service, Finance, Billing and Accounting']\n",
      "set2_Chicago                             # \n",
      "Miller Cook response locations           # shared # ['Demographic Content']\n",
      "ID                                       # \n",
      "set1_fortune500                          # \n",
      "set1_gov_sites_registration              # \n",
      "Smith Collins response locations         # shared # ['Central Services']\n",
      "set1_india                               # \n",
      "Johnson Stewart response locations       # shared # ['Central Services']\n",
      "set2_SD_crime                            # \n",
      "IN                                       # \n",
      "set1_GeoJson                             # \n",
      "LA                                       # \n",
      "Moore Bell response locations            # shared # ['Compliance', 'Demographic Content']\n",
      "set2_empty                               # \n",
      "Williams Sanchez response locations      # shared # ['Customer Service, Finance, Billing and Accounting']\n",
      "NH                                       # \n",
      "IN                                       # \n",
      "AR                                       # \n",
      "AZ                                       # \n",
      "Wilson Morgan response locations         # shared # ['Compliance', 'Demographic Content']\n",
      "Davis Reed response locations            # shared # ['Demographic Content']\n",
      "Smith Collins response locations         # \n",
      "NC                                       # \n",
      "set1_mapping_tech                        # \n",
      "USA_cities_Fortune_500                   # \n",
      "Taylor Murphy response locations         # shared # ['Central Services', 'Compliance']\n",
      "set2_Voronoi-diagram                     # \n",
      "set1_major_cities                        # \n",
      "LA                                       # \n",
      "NH                                       # "
     ]
    }
   ],
   "source": [
    "for key in source_items_by_id.keys():\n",
    "    item = source_items_by_id[key]\n",
    "    print(\"\\n{:40s}\".format(item.title), end = \" # \")\n",
    "    if 'groups' in item:\n",
    "        print(item.access, end = \" # \")\n",
    "        print(item.groups, end = \"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see from above, some items are shared to a few groups while some are not."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "source": [
    "## Copy Items\n",
    "Below we define a function that you can call in a loop for each item in the dictionary we composed earlier. If the item is a text based item such as a Web Map or a file based item such as a layer package, it downloads the item's data to a temporary directory and uses that for creating the target item during cloning. You can find the [exhaustive list of different items](https://doc.arcgis.com/en/arcgis-online/reference/supported-items.htm) that you can upload to your portal and their corresponding item types from the [REST API documentation](https://developers.arcgis.com/rest/users-groups-and-items/items-and-item-types/). For brevity, this sample covers only a subset of those items. Note, if the item points to a web layer URL, the target item would also point to the same URL."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "TEXT_BASED_ITEM_TYPES = frozenset(['Web Map', 'Feature Service', 'Map Service','Web Scene',\n",
    "                                   'Image Service', 'Feature Collection', \n",
    "                                   'Feature Collection Template',\n",
    "                                   'Web Mapping Application', 'Mobile Application', \n",
    "                                   'Symbol Set', 'Color Set',\n",
    "                                   'Windows Viewer Configuration'])\n",
    "\n",
    "FILE_BASED_ITEM_TYPES = frozenset(['File Geodatabase','CSV', 'Image', 'KML', 'Locator Package',\n",
    "                                  'Map Document', 'Shapefile', 'Microsoft Word', 'PDF',\n",
    "                                  'Microsoft Powerpoint', 'Microsoft Excel', 'Layer Package',\n",
    "                                  'Mobile Map Package', 'Geoprocessing Package', 'Scene Package',\n",
    "                                  'Tile Package', 'Vector Tile Package'])\n",
    "\n",
    "ITEM_COPY_PROPERTIES = ['title', 'type', 'typeKeywords', 'description', 'tags',\n",
    "                        'snippet', 'extent', 'spatialReference', 'name',\n",
    "                        'accessInformation', 'licenseInfo', 'culture', 'url']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We define the copy function for items below. This function gets the properties of the item from source and applies it to the target. If the items were saved inside a folder, it creates that folder on the target as well. Finally, it sets the privacy (sharing) properties similar to how it was on the source portal."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "def copy_item(target, source_item):\n",
    "    try:\n",
    "        with tempfile.TemporaryDirectory() as temp_dir:\n",
    "            item_properties = {}\n",
    "            for property_name in ITEM_COPY_PROPERTIES:\n",
    "                item_properties[property_name] = source_item[property_name]\n",
    "\n",
    "            data_file = None\n",
    "            \n",
    "            if source_item.type in TEXT_BASED_ITEM_TYPES:\n",
    "                # If its a text-based item, then read the text and add it to the request.\n",
    "                text = source_item.get_data(False)\n",
    "                item_properties['text'] = text\n",
    "            \n",
    "            elif source_item.type in FILE_BASED_ITEM_TYPES:\n",
    "                # download data and add to the request as a file\n",
    "                data_file = source_item.download(temp_dir)\n",
    "\n",
    "            item_properties[\"thumbnail\"] = source_item.download_thumbnail(temp_dir)\n",
    "            item_properties[\"metadata\"] = source_item.download_metadata(temp_dir)\n",
    "\n",
    "            #find item's owner\n",
    "            source_item_owner = source.users.search(source_item.owner)[0]\n",
    "            \n",
    "            #find item's folder\n",
    "            item_folder_titles = [f['title'] for f in source_item_owner.folders \n",
    "                                  if f['id'] == source_item.ownerFolder]\n",
    "            folder_name = None\n",
    "            if len(item_folder_titles) > 0:\n",
    "                folder_name = item_folder_titles[0]\n",
    "\n",
    "            #if folder does not exist for target user, create it\n",
    "            if folder_name:\n",
    "                target_user_folders = target.content.folders.get(folder_name)\n",
    "                if target_user_folders is None:\n",
    "                    #create the folder\n",
    "                    folder = target.content.folders.create(folder_name, source_item.owner)\n",
    "            else:\n",
    "                    folder = target.content.folders.get() #root folder\n",
    "            \n",
    "            # Add the item to the target portal, assign owner and folder\n",
    "            target_item = folder.add(item_properties, data_file)\n",
    "            \n",
    "            #Set sharing (privacy) information\n",
    "            share_everyone = source_item.access == 'public'\n",
    "            share_org = source_item.access in ['org', 'public']\n",
    "            share_groups = []\n",
    "            if source_item.access == 'shared':\n",
    "                share_groups = source_item.groups\n",
    "            \n",
    "            target_item.share(share_everyone, share_org, share_groups)\n",
    "            \n",
    "            return target_item\n",
    "        \n",
    "    except Exception as copy_ex:\n",
    "        print(\"\\tError copying \" + source_item.title)\n",
    "        print(\"\\t\" + str(copy_ex))\n",
    "        return None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copy over each item. While doing so, construct a dictionary mapping of source item's ID with target item's ID"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Copying KS \tfor\t smith.collins\n",
      "Copying NC \tfor\t jones.morris\n",
      "Copying AR \tfor\t brown.rogers\n",
      "Copying set2_catalina-points \tfor\t project_archiver\n",
      "Copying FL \tfor\t davis.reed\n",
      "Copying KS \tfor\t smith.collins\n",
      "Copying set1_GeoJson \tfor\t project_archiver\n",
      "Copying set2_australia \tfor\t project_archiver\n",
      "Copying NV \tfor\t johnson.stewart\n",
      "Copying AZ \tfor\t moore.bell\n",
      "Copying NV \tfor\t johnson.stewart\n",
      "Copying FL \tfor\t davis.reed\n",
      "Copying set3_Streets \tfor\t project_archiver\n",
      "Copying set2_counties \tfor\t project_archiver\n",
      "Copying ID \tfor\t wilson.morgan\n",
      "Copying Brown Rogers response locations \tfor\t brown.rogers\n",
      "Copying set1_Chicago \tfor\t project_archiver\n",
      "Copying set2_USAcities \tfor\t project_archiver\n",
      "Copying Jones Morris response locations \tfor\t jones.morris\n",
      "Copying set2_Chicago \tfor\t project_archiver\n",
      "Copying Miller Cook response locations \tfor\t miller.cook\n",
      "Copying ID \tfor\t wilson.morgan\n",
      "Copying set1_fortune500 \tfor\t project_archiver\n",
      "Copying set1_gov_sites_registration \tfor\t project_archiver\n",
      "Copying Smith Collins response locations \tfor\t smith.collins\n",
      "Copying set1_india \tfor\t project_archiver\n",
      "Copying Johnson Stewart response locations \tfor\t johnson.stewart\n",
      "Copying set2_SD_crime \tfor\t project_archiver\n",
      "Copying IN \tfor\t williams.sanchez\n",
      "Copying set1_GeoJson \tfor\t project_archiver\n",
      "Copying LA \tfor\t taylor.murphy\n",
      "Copying Moore Bell response locations \tfor\t moore.bell\n",
      "Copying set2_empty \tfor\t project_archiver\n",
      "Copying Williams Sanchez response locations \tfor\t williams.sanchez\n",
      "Copying NH \tfor\t miller.cook\n",
      "Copying IN \tfor\t williams.sanchez\n",
      "Copying AR \tfor\t brown.rogers\n",
      "Copying AZ \tfor\t moore.bell\n",
      "Copying Wilson Morgan response locations \tfor\t wilson.morgan\n",
      "Copying Davis Reed response locations \tfor\t davis.reed\n",
      "Copying Smith Collins response locations \tfor\t smith.collins\n",
      "Copying NC \tfor\t jones.morris\n",
      "Copying set1_mapping_tech \tfor\t project_archiver\n",
      "Copying USA_cities_Fortune_500 \tfor\t project_archiver\n",
      "Copying Taylor Murphy response locations \tfor\t taylor.murphy\n",
      "Copying set2_Voronoi-diagram \tfor\t project_archiver\n",
      "Copying set1_major_cities \tfor\t project_archiver\n",
      "Copying LA \tfor\t taylor.murphy\n",
      "Copying NH \tfor\t miller.cook\n"
     ]
    }
   ],
   "source": [
    "source_target_itemId_map = {}\n",
    "for key in source_items_by_id.keys():\n",
    "    source_item = source_items_by_id[key]\n",
    "\n",
    "    print(\"Copying {} \\tfor\\t {}\".format(source_item.title, source_item.owner))\n",
    "    target_item = copy_item(target, source_item)\n",
    "    if target_item:\n",
    "        source_target_itemId_map[key] = target_item.itemid\n",
    "    else:\n",
    "        source_target_itemId_map[key] = None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have successfully cloned all the items from source to target. We can query the contents of one of the users below to verify:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div class=\"9item_container\" style=\"height: auto; overflow: hidden; border: 1px solid #cfcfcf; border-radius: 2px; background: #f6fafa; line-height: 1.21429em; padding: 10px;\">\n",
       "                    <div class=\"item_left\" style=\"width: 210px; float: left;\">\n",
       "                       <a href='https://dev003735.esri.com/portal/home/user.html?user=brown.rogers' target='_blank'>\n",
       "                        <img src='' width='200' height='133' class=\"itemThumbnail\">\n",
       "                       </a>\n",
       "                    </div>\n",
       "\n",
       "                    <div class=\"item_right\" style=\"float: none; width: auto; overflow: hidden;\">\n",
       "                        <a href='https://dev003735.esri.com/portal/home/user.html?user=brown.rogers' target='_blank'><b>Brown Rogers</b>\n",
       "                        </a>\n",
       "                        <br/><br/><b>Bio</b>: None\n",
       "                        <br/><b>First Name</b>: Brown\n",
       "                        <br/><b>Last Name</b>: Rogers\n",
       "                        <br/><b>Username</b>: brown.rogers\n",
       "                        <br/><b>Joined</b>: April 04, 2017\n",
       "\n",
       "                    </div>\n",
       "                </div>\n",
       "                "
      ],
      "text/plain": [
       "<User username:brown.rogers>"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "user1 = target.users.search()[2]\n",
    "user1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<Item title:\"AR\" type:Feature Service owner:brown.rogers>,\n",
       " <Item title:\"AR\" type:CSV owner:brown.rogers>]"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "user1.items()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We could query the folders belonging to this user and the items within as well"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'created': 1491335086184,\n",
       "  'id': 'ebfdbc2bb47f4d5f92b6673a8a68a89f',\n",
       "  'title': 'Rogers_webmaps',\n",
       "  'username': 'brown.rogers'}]"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "user1.folders"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<Item title:\"Brown Rogers response locations\" type:Web Map owner:brown.rogers>]"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "user1.items(folder=user1.folders[0]['title'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Establish relationship between items\n",
    "\n",
    "So far, we have successfully cloned users, groups and items from source to target. Next, we will establish identical [relationships](https://developers.arcgis.com/rest/users-groups-and-items/relationship-types/) between items as they were in the source portal."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "RELATIONSHIP_TYPES = frozenset(['Map2Service', 'WMA2Code',\n",
    "                                'Map2FeatureCollection', 'MobileApp2Code', 'Service2Data',\n",
    "                                'Service2Service'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below, we loop through each item in source portal, find to which other item it is related and the type of that relationship. If a relationship is found, we find the corresponding items in target and establish the same relationship. To make this work, we will make use of the dictionary that maps the itemIds on source and target we created during the item clone stage. Let us take a look at that dictionary below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'0c8ffe4ab7754eedbd2c514032f7e913': 'a83e41e4768c486d8fbece1a9cd819b1',\n",
       " '0fdcdd6eed9e4e6aa83575da5c4d8ff0': 'b7b266c064634c10868cae2daf6b922e',\n",
       " '10eff50fda644f2fa9261ee925a24495': '4328e2087aa74f92913e2af005d7590e',\n",
       " '14935a49a8544d8eb0eeadaa6216bbba': '034c8ccc02de4d199171ad5ac53decbe',\n",
       " '1506548f309140d89a19886913d8395a': '1a167bb7f305456087db833a956e0c8a',\n",
       " '1cb04f79f8a74826bab704efe5206603': 'fa725ccf2c31446d99efe6792403564d',\n",
       " '1d8820ef73064498983af1af326ddb9d': '66c5fe735bf142f0b0d79e1de0372a45',\n",
       " '28984d7a69f34b51b07a087800931d6f': 'a3e4a6ea11124ff48fa6f13100649756',\n",
       " '297a1819ba9b43ffb8adcab6a7eebd53': 'ef11e8a203014cb4899c1c61369437c6',\n",
       " '464fdece247e46b1ba861a7ac92701a4': '95f05a2413e8471c9fa06ed4b9f65f6c',\n",
       " '48f17e7559ca4d65b5eab5b78cabf085': 'be9e963a42ea4ae4872bc12059db3eeb',\n",
       " '54cf94db12e64092bef153b9f2c07176': 'ba525e9a038748dd9fe595151a7c5c73',\n",
       " '56d1921b6cfb46e89bd697668cceb708': '3db062fa1aa249efaac1ed734edf8779',\n",
       " '59132371ac22462a899bcaf7b561f1cd': '93a4bf9d217349fb9324eb2fafda5621',\n",
       " '67503e5077f240738e091e4c1da13745': 'e256401189684e1ba88ac175d01ff2d6',\n",
       " '715f8038c23b4b53a1280ae1b7b4a9bc': 'd2a560bff4034238b8b2b5278e7855ff',\n",
       " '71b66b8e628b46c9850cefd91f67f604': '18861c8f7833430fa621f566da8d055a',\n",
       " '73118cf2a8334a9884f269f81ab25c45': 'a845548e6de94b179ffab40819b8a175',\n",
       " '734529a8daad4317bc6daedfe01fdc9f': '90e2b02043df4d6a818cd3c11d9060ff',\n",
       " '75c63c8de25943f9b5ed9c477e556034': 'ae666900d38d4b0c8a610bfdc1af74c5',\n",
       " '7be7710b5e4b44cf836139a115825b2e': '6918e98589a846e7877e08a2678c52bc',\n",
       " '7c4fdc5508004253960993bbab02bf61': 'd7ded032276244cb93da101eed2a6de9',\n",
       " '7e854deebcd1442eb850a3ea84732679': '8f43bca0d1c04ab1939d7840d27d497b',\n",
       " '819d0cbed11f4f81a98e7f388fb8eeab': '9819e9b97e6b45f7b472fcea3813e9a7',\n",
       " '882009025e11471caad3f48389fde221': 'c76bab88f25843d498500b13593bc5cb',\n",
       " '8a6b1ed1b7be4cdeb298020cf5108c70': '60158b3293de4b17bd9dab7a5e49823b',\n",
       " '8d025e7368974649a3eb0691656dfa57': 'fe174f158e46437a91bd5e1bcd1208f9',\n",
       " '97d15ea9057549cdbe67344fe129c758': 'a174cddd5c2e4795a8cb3c24e3923917',\n",
       " '9afed1a827914a53af34af490f64ecdf': 'ec6adb1ba69c4b62a0a7ffc5bb77b8d0',\n",
       " 'a0929a38db3240cfaffbc254d00e1827': '2544da801896434cbdee71ba53595569',\n",
       " 'a89398de64444d58a9ea6a0cc407c6b7': '3bd28b9286d3406b89624d5156f42bc1',\n",
       " 'aaa6ad38f9c24b4294a4c6d0994fc7c7': '985ba4a4fca9425ebdc7599ed29f4673',\n",
       " 'b18c614bc00d40d0a216754d1aa8b150': 'b442463bd44341ad95858c8f5c386714',\n",
       " 'b8c648204c804c289930b9e121dd8fdb': 'a90f58b560ea421193e63fddccc93b42',\n",
       " 'ba3d9b3e6793406696497be34607de44': 'd2a2aa9425854b24a561b282632fde56',\n",
       " 'c56e60ab29f1400fbc46cc2732724ef8': '1200541fdc444a9680e5bc447f0fb23b',\n",
       " 'd65f34a670654818a010771d9a896760': '8f02739994df48e795b2b8a2b1e09dc6',\n",
       " 'd6818b3d16c847b5ab9986d708b992a2': '457041ce43564d7e8304d128787bcbdb',\n",
       " 'd8fdaa873f724c7abdca62a42de0c97a': '38bf01476282444e9f3468324fde8743',\n",
       " 'dee8ae7fede64f8e94c39d4f6aff8d6f': '5b185e3231224f37af416f2c55d6747d',\n",
       " 'e0e716ac1d774cb190765dd7a2a7421f': 'e5099e2a674f49aebf4a140920abf67a',\n",
       " 'e76f3fab66534499bdc394558a645357': '9be7de598e714b66ab2058d85313e69f',\n",
       " 'ea36c5dbd4e142079c43b17cb49b4258': '390da582debf43ee8dd5a2a17df816b1',\n",
       " 'ebd57622fb984f1c96122c4b442ea6d7': '9f9f91d7a0e64072921f5eced954168c',\n",
       " 'edba588da56b4a3ab2f2df576f9c6b99': 'e0f42fc1c8c341b9925debace5e402f9',\n",
       " 'eed82e9d95dc4ad6a0f3d21e530e0a8f': '2efdd82ec6c84e9390f304b5c3240081',\n",
       " 'f3f87cb385ea4ae291fd6557338e8e0b': '6fe88968317a45f9830cb9177df75c13',\n",
       " 'f9866c5a2be0428abe1c7097993d9b36': '960347c4c3764e96bc15ca2817752563'}"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "source_target_itemId_map"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NC # Feature Service\n",
      "\t\tNC # CSV\t## Service2Data\n",
      "\t\testablishing relationship in target portal True\n",
      "AR # Feature Service\n",
      "\t\tAR # CSV\t## Service2Data\n",
      "\t\testablishing relationship in target portal True\n",
      "set2_catalina-points # KML\n",
      "FL # Feature Service\n",
      "\t\tFL # CSV\t## Service2Data\n",
      "\t\testablishing relationship in target portal True\n",
      "KS # Feature Service\n",
      "\t\tKS # CSV\t## Service2Data\n",
      "\t\testablishing relationship in target portal True\n",
      "set1_GeoJson # PDF\n",
      "set2_australia # GeoJson\n",
      "Smith Collins response locations # Web Map\n",
      "AZ # Feature Service\n",
      "\t\tAZ # CSV\t## Service2Data\n",
      "\t\testablishing relationship in target portal True\n",
      "NV # Feature Service\n",
      "\t\tNV # CSV\t## Service2Data\n",
      "\t\testablishing relationship in target portal True\n",
      "FL # CSV\n",
      "set3_Streets # Map Document\n",
      "set2_counties # Locator Package\n",
      "ID # Feature Service\n",
      "\t\tID # CSV\t## Service2Data\n",
      "\t\testablishing relationship in target portal True\n",
      "Wilson Morgan response locations # Web Map\n",
      "set1_Chicago # CSV\n",
      "set2_USAcities # File Geodatabase\n",
      "Jones Morris response locations # Web Map\n",
      "set2_Chicago # CSV\n",
      "Miller Cook response locations # Web Map\n",
      "ID # CSV\n",
      "set1_fortune500 # File Geodatabase\n",
      "set1_gov_sites_registration # Microsoft Excel\n",
      "NV # CSV\n",
      "set1_india # GeoJson\n",
      "Johnson Stewart response locations # Web Map\n",
      "set2_SD_crime # Map Document\n",
      "IN # Feature Service\n",
      "\t\tIN # CSV\t## Service2Data\n",
      "\t\testablishing relationship in target portal True\n",
      "set1_GeoJson # Microsoft Word\n",
      "LA # Feature Service\n",
      "\t\tLA # CSV\t## Service2Data\n",
      "\t\testablishing relationship in target portal True\n",
      "Moore Bell response locations # Web Map\n",
      "set2_empty # Map Document\n",
      "set1_major_cities # Locator Package\n",
      "NH # CSV\n",
      "AZ # CSV\n",
      "AR # CSV\n",
      "IN # CSV\n",
      "Brown Rogers response locations # Web Map\n",
      "Davis Reed response locations # Web Map\n",
      "Smith Collins response locations # Web Map\n",
      "NC # CSV\n",
      "set1_mapping_tech # Microsoft Powerpoint\n",
      "USA_cities_Fortune_500 # Map Document\n",
      "Taylor Murphy response locations # Web Map\n",
      "set2_Voronoi-diagram # Microsoft Word\n",
      "Williams Sanchez response locations # Web Map\n",
      "LA # CSV\n",
      "NH # Feature Service\n",
      "\t\tNH # CSV\t## Service2Data\n",
      "\t\testablishing relationship in target portal True\n"
     ]
    }
   ],
   "source": [
    "for key in source_target_itemId_map.keys():\n",
    "    source_item = source_items_by_id[key]\n",
    "    target_itemid = source_target_itemId_map[key]\n",
    "    target_item = target.content.get(target_itemid)\n",
    "\n",
    "    print(source_item.title + \" # \" + source_item.type)\n",
    "    for relationship in RELATIONSHIP_TYPES:\n",
    "        try:\n",
    "            source_related_items = source_item.related_items(relationship)\n",
    "            for source_related_item in source_related_items:\n",
    "                print(\"\\t\\t\" + source_related_item.title + \" # \" + \n",
    "                      source_related_item.type +\"\\t## \" + relationship)\n",
    "\n",
    "                #establish same relationship amongst target items\n",
    "                print(\"\\t\\t\" + \"establishing relationship in target portal\", end=\" \")\n",
    "                target_related_itemid = source_target_itemId_map[source_related_item.itemid]\n",
    "                target_related_item = target.content.get(target_related_itemid)\n",
    "                status = target_item.add_relationship(target_related_item, relationship)\n",
    "                print(str(status))\n",
    "        except Exception as rel_ex:\n",
    "            print(\"\\t\\t Error when checking for \" + relationship + \" : \" + str(rel_ex))\n",
    "            continue"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "Thus, with this notebook, we have successfully cloned groups, users and their contents. Note, this notebook did not copy over the services that power the service based items. Such items continue to point to the same URL as the ones in source portal did. As long as those URLs remain accessible, the web maps and layer items continue to be usable.\n",
    "\n",
    "To run this notebook as a Python script, check out the [ArcGIS API for Python Public Repo scripts](https://github.com/Esri/arcgis-python-api/tree/master/samples/03_org_administrators)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.0"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": true,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
